1. Field of the Invention
The present invention relates to an image processing apparatus and method.
2. Description of the Related Art
With the progress that has been made in the digitization of information in recent years, there has been a proliferation of systems in which paper documents, rather than being archived in paper form, are scanned in as by a scanner and stored in electronic form or transmitted to another apparatus in the form of electronic data. In order to reduce transmission cost, a high degree of data compression is required for documents in electronic form. Also required, on the other hand, are the ability to partially edit electronic data and high image quality that will not decline regardless of whether the data is enlarged or reduced in size.
There are certain problems, however. Specifically, in a case where a document image contains a mixture of text and photographic regions, image quality is good but compression rate declines if compression suited to a text region is applied, whereas compression rate is high but text is degraded if compression suited to a photographic region is applied. Accordingly, in a technique proposed heretofore, a document image in electronic form is separated into text and photographic regions, the text region is subjected to compression suited to the text region, the photographic region is subjected to compression suited to the photographic region after text pixels are filled in with pixel values of surrounding pixels, and the compressed text image and background image are output together (e.g., see the specifications of Japanese Patent Laid-Open Nos. 07-236062 and 2005-012768).
In another proposed technique, a text region in which excellent reutilization and high image quality are important is converted to vector data, other regions such as photographic regions that do not readily lend themselves to reproduction by vectorization are compressed according to the JPEG standard, and the results of compressing these regions are combined and output, thereby realizing high compression, excellent reutilization and high image quality of document images (e.g., see the specification of Japanese Patent Laid-Open No. 2004-265384).
In a further proposed technique, targets of vector processing are expanded beyond just text and line drawings to thereby improve the ability to compress and reutilize document images and raise the image quality thereof (e.g., see the specification of Japanese Patent Laid-Open No. 2006-344069). This technique vectorizes specific images (e.g., illustrations) characterized by an object outline (object boundary) that is clearer and colors more limited in comparison with a natural image such as a photograph. Such a specific image is referred to as a “clipart image”.
Further, in the specification of Japanese Patent Laid-Open No. 2007-305034, regions capable of being vectorized in a document image containing text regions are vectorized and the result of vectorization is placed in a vector layer. Vectorized regions are filled in with the neighboring background pixel values, the image obtained by such filling is subjected to JPEG compression and the result of compression is placed in a JPEG layer. In this way a document file comprising the vector-layer data and the JPEG-layer data is created.
Further, the specification of Japanese Patent Laid-Open No. 2004-128881 proposes a method in which a binary image having the original resolution and a multi-valued image having half the original resolution are generated from an input image, the binary image is subjected to separation processing and text-region conversion processing, and the multi-valued image is subjected to background-region conversion processing. For example, in a case where the input image is size A4 and consists of about 24 MB of data at 24 pits per pixel, the binary image obtained by binarizing the input image will consist of about 1 MB of data, and the image obtained by halving the resolution of the input image will consist of about 6 MB of data. By discarding the original input image after these two images are generated, digitization processing can be executed using an image memory of approximately one-fourth the size.
It is conceivable that the problem set forth below will arise in a case where a clipart region is defined by a rectangular area, the clipart region in a document image is vectorized and then fill-up is performed using the surrounding pixel values.
By way of example, when a clipart region (illustration region) 101 of the kind shown in FIG. 1 is represented by a rectangle, there are instances where another region (photographic region 102) overlaps the background portion of the clipart region. If the entire rectangular clipart region 101 is filled in with the surrounding pixel values, another region 201 that overlaps the clipart region 101 vanishes owing to such fill-up, as illustrated in FIG. 2. At the time of image reproduction, the image of the photographic region 102 cannot be reproduced. For this problem, the present applicant considers a method as below, the method being proposed in the specification of U.S. patent application Ser. No. 12/432,261. In this method, the entire rectangle of the clipart region 101 is not filled in. Rather, the region within the clipart region 101 is divided into a background region and an area other than background [i.e., an object region (the illustration proper) that constitutes foreground]. Then, only the region other than the background [namely the object region (foreground region)] is filled in.
In accordance with this method, a synthesized document file having a high compressibility can be generated and the original image can be reproduced also in a case where a region overlapping the periphery exists.
In the example of the document image shown in FIG. 1, fill-up processing is applied only to the object region (the illustration proper) contained in the clipart region 101, as shown in FIG. 3. That is, the region of the clipart region 101 other than background (namely the object region) is filled in. As a result, the overlapping portion of the photograph is entirely unaffected.
In this example, the document image that includes the clipart region 101 is input, the document image thus input is converted to binary image data and the clipart region is extracted by executing region separation processing which, using the binary image data, separates the image into various types of regions such as text, graphics and tables. Furthermore, on the basis of the color of the document image corresponding to the clipart region, the clipart region is segmented into a plurality of regions based upon degree of color similarity. Based upon the results of such region segmentation, a region that should be construed as the background region is specified within the clipart region. Then, in this document image, the color information in the region to be construed as background is adopted as the background color and the object region (the region other than background) within the clipart region is filled in with the background color. On the other hand, the image of the object region within the clipart region is converted to vector data of the clipart. The document image that results from filling in the object region (the region other than background) is subjected to JPEG compression as a base image and the JPEG data of the compressed base image is output in a file format in which the vector data of the clipart region is rendered on the base image.
Thus, in the proposed document image processing method, even in a document image that has a clipart region and another overlapping region, the other region that overlaps the clipart region will not vanish owing to fill-up. Further, it is possible to reproduce the original image and improve compressibility, reutilization and image quality.
On the other hand, in a case where an image processing apparatus is equipped with such processing, a large memory is required if the base image to be filled in is stored as is. This raises the cost of the apparatus. In the present invention, therefore, an image of interest is processed upon converting the color image format in RGB color space obtained by opto-electronic scanning that is based upon the three primary colors of the scanner. For example, a conversion is made to a format expressed in color space divided into a brightness component (Y) and color components (Cr, Cb) as in the manner of well-known YCrCb color space. In order to apply vectorization processing to an input image the size of which is large in comparison with the limitation in memory capacity permitted by the system, the resolution of the base image is lowered. For example, when the brightness component and color components of the input image have a ratio indicated by Y:Cr:Cb=4:4:4, the resolution of the color components (Cr, Cb) of the base image is halved. As a result, a conversion is made in such a manner that the ratio of the numbers of pixels becomes Y:Cr:Cb=4:1:1 so that the amount of data necessary for the base image is reduced. It should be noted that with regard to the object to undergo vectorization (an object portion of the clipart region, etc.), processing is executed at the resolution of the input image.
By adopting this expedient, a base image can be handled with half the amount of data in comparison with a case where a color image is stored, without lowering resolution, in the RGB format based upon the three primary colors. In other words, image data having twice the size can be processed. It should be noted that the reason for lowering the resolution of Cr, Cb more than the resolution of the brightness component Y is that the characteristic of the human eye is such that the eye is more sensitive to the brightness component Y than to the color components Cr, Cb.
With such an arrangement, however, there are cases where a false color based upon a difference in the resolutions of color components and fill-up color occurs at rectangular boundary portions and at background boundary portions, and this can be a factor that detracts from the image quality of the output image. It should be noted that a rectangular boundary portion is the boundary between an object that is to be vectorized and the outside of the rectangular area circumscribing the object. For example, it is the boundary between an object region within a rectangular clipart region and an base image outside the clipart region. Further, a background boundary portion is the boundary between an object to be vectorized and a background region other than the object to be vectorized inside the circumscribing rectangle. For example, it is the boundary between an object region inside a rectangular clipart region and the background region.
The cause of a false color that occurs at the above-mentioned boundary portions will be described with reference to FIGS. 4 to 6. FIG. 4 is a diagram illustrating the brightness component (Y component) of a color image, FIG. 5 is a diagram illustrating a color component (Cr component) of a color image, and FIG. 6 is a diagram illustrating a color component (Cb component) of a color image. FIG. 4 represents the Y component when the resolutions of the Cr. Cb components have been made lower than the resolution of the Y component in such a manner that Y:Cr:Cb=4:1:1 will hold. FIGS. 5 and 6 represent the Cr and Cb components, respectively, in a similar manner.
Reference numerals 401 in FIG. 4, 501 in FIG. 5 and 601 in FIG. 6 represent the Cr component and Cb component corresponding to a 2×2 pixel region at the resolution of the Y component in image data having the Y:Cr:Cb=4:1:1 format in YCrCb color space. In other words, the 2×2 pixel region (Y11, Y12, Y13, Y14) of the Y component corresponds to one pixel in the Cr component and Cb component.
Further, reference numerals 402 in FIG. 4, 502 in FIG. 5 and 602 in FIG. 6 are examples indicating the positions at which an object to be vectorized in the entered color image (original image) exists. In the case of the Y component whose resolution is the same high resolution as that of the original image, the shape of the object of interest agrees with the pixel boundary. On the other hand, in the case of the Cr component and Cb component whose resolution is half that of the original image, the shape of the object of interest does not necessarily agree with the pixel boundaries of the color components Cr, Cb. That is, there are cases where the boundary portion of the object of interest partially overlaps an area that is one-fourth or one-half of a pixel in the color components Cr, Cb.
Such color components (Cr, Cb) of the boundary portion can be unified with the color information on the object side, or they can be unified with the color information on the base image side, or the color information on the object side and the color information of the base image can be made an average color that takes into consideration the partially overlapping area ratio.
If under these conditions a conversion is made to the RGB format, which is based upon the three primary colors having the same resolution for each color, in order to display the image on a display unit or print the image by a printer, this will cause a false color to appear at the boundary portions regardless of which of the above-described methods is adopted for processing. In other words, the pixel-boundary color information is such that a color (a false color) is produced at the boundary portions that is different from the color of the color image in RGB color space obtained by opto-electronic scanning based upon the three primary colors of the original scanner. The false color results in an unnatural appearance.
There is the possibility that such false color will be occur at least at one or both boundary portions, namely the boundary portion between a background region within a clipart region and the object region (the region other than background region) and the boundary portion between the clipart region and the base region. This is illustrated in FIGS. 7A to 7G.
FIGS. 7A to 7G are diagrams useful in describing a false color that occurs at the boundary portion between a background region and an object region and at the boundary portion between an object region and a base region. FIG. 7A represents an example of a document image that contain a clipart region. Reference numerals 701 indicate the position of a clipart region that is to be vectorized. It indicates an object-circumscribing rectangle comprising a rectangle circumscribed around the clipart region. FIG. 7B represents a state in which the image within the object-circumscribing rectangle 701 has been extracted. Reference numerals 702 denote background within the clipart region (within the object-circumscribing rectangle). The background is specified as a background region from the result of segmenting the clipart region into a plurality of regions based upon degree of color similarity. The method used to specify the background region includes clustering images inside the clipart region based upon similar colors, and adopting, as a cluster of the background region, the cluster having the largest number of pixels contiguous to the boundary of the circumscribed rectangle of the clipart region. Reference numerals 704 in FIG. 7A denote the base image.
FIG. 7C excludes a background region 703 from the vectorization target and represents only a region other than background (i.e., only the object to be vectorized). It should be noted that the background color is found from the background 702 corresponding to the background region 703. Reference numerals 705 in FIG. 7C denote a boundary portion between the object region (the region other than background) and the base image 704, and reference numerals 706 denote a boundary portion between the object region (the region other than background) and the background 702. Further, FIG. 7D illustrates a region 707, which is other than background, that is to be filled in the base image. FIG. 7E illustrates a state in which both the boundary portion 705 and the boundary portion 706 have been filled in with the background color, which has been found from the background 702, without distinguishing them from the region 707 other than background.
In FIG. 7F, the object portion inside the clipart region is filled in with the background color, as illustrated in FIG. 7E, in the base image, and the result of vectorizing the object portion of the clipart region is displayed in superimposed form. The boundary portion 706 does not present a problem but there is large difference between the filled-in color (the background color) and the color of the base image at the portion corresponding to the boundary portion 705, as indicated at 708. When the vectorized object portion is displayed in superimposed form, a false color is produced owing to the difference between resolutions of the object portion and base image.
On the other hand, if both the boundary portion 705 and the boundary portion 706 are filled in not with the background color but with the base color, now a false color will not be produced in relation to the boundary portion 705 with the base image but a false color will be produced in relation to the boundary portion 706 with the background region.
A first object of the present invention is to prevent the occurrence of a false color in an object image, which is contained in a color image, and the boundary with this object image.
Furthermore, if an image used in region analysis processing (region separation processing) or vector conversion processing, etc., does not possess enough resolution and color-gamut information, etc., to maintain the accuracy of each of these types of processing, it will not be possible to generate electronic data having the advantages of high image quality and high reutilization. On the other hand, the amount of information required in each of these types of processing differs. If an image containing a large amount of information is input to all such types of processing uniformly, extra memory is consumed and processing speed slows down.
For example, in a case where a high image quality is required for a character portion in electronic data to be generated, it is necessary to input an image having a higher resolution for text-region conversion processing. For example, in order to generate vector data, which is for reproducing appearance faithfully, from an image of a small character having a point size of 3 to 5 points, an image having a high resolution on the order to 600 dpi is necessary. On the other hand, if an image for region analysis processing (in which the interior of an image is separated into a text region, a clipart region and a photographic region, etc.) has a resolution of 300 dpi, this will be sufficient. A resolution higher than this may not only invite an increase in amount of memory used and a decline in processing speed but may also lower processing accuracy because of an increase in amount of noise. Further, since it will suffice if a region can be segmented, it is more advantageous in view of raising processing speed if quantization has been performed by pre-processing so as to obtain two values locally and at most N values even for the entire image. Furthermore, with respect to processing for vectorizing a clipart region, a resolution as high as that for processing a text region is not required and an image of intermediate resolution on the order of 300 dpi is suitable. Further, in the results of vectorization, the number of bits per pixel in the input image must be made sufficiently large in order to reproduce colors that agree visually with those of the original. However, if it is so arranged that the color differences Cb, Cr have a resolution that is one-half that of luminance Y, as described above, then the size of the image data can be suppressed.